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Abstract 

This study was designed primarily to investigate the impact of anonymous peer feedback on student 
meaningful learning in higher education. Forty-seven students from three undergraduate classes from a central 
US university participated in this study. Students were asked to build a web-based project. In the experimental 
group , technology-mediated peer review and feedback were provided for students to use in improving then- 
projects prior to instructor assessment. The control group received no peer feedback. Students’ projects were 
independently evaluated and analyzed. Results indicated that there was no significant difference on project 
quality between the control and experimental groups. However, post-assessment survey indicated that students 
had generally positive perceptions of this process. 

Introduction 

Promoting student autonomy and encouraging student meaningful learning has become an important 
focus in higher education in recent years. When students take a more active role, learning becomes more 
meaningful and their achievement is improved. Researchers (Orsmond & Merry, 1996; Orsmond, Merry, & 
Reiling, 2002) argued the need for academic staff to switch their roles from teaching to facilitating learning in 
order to achieve higher student engagement and responsibility and suggested that in assessment practices, some 
“power” should be “handed over” to students. 

Students’ behavior and attitude toward learning are shaped by the assessment system (Freeman, 1995). 
To achieve the outcome of meaningful learning, appropriate assessment methods should be applied. 
Unfortunately, the traditional instructor-led assessment method provides only limited opportunities for 
assessment and feedback. Peer assessment is believed to be one of the solutions, as it not only provides 
additional feedback but also stimulates student interaction and involves students in thinking critically about 
assessment criteria. Within this context, the assessment process can be viewed as “the learning exercise in 
which the assessment skills are practiced.” (Sluijsmans, Brand-Gruwel, & van Merrienboer, 2002). 

Peer assessment, according to Topping and his colleagues (Topping, Smith, Swanson, & Elliot, 2000), is a 
process in which peers evaluate the achievement or performance of others of similar status. Cheng & Warren 
(Cheng & Warren, 1999) further defined this assessment form as reflection on “what learning had taken place 
and how.” Peer assessment, as an alternative to traditional solo instructor assessment, has been applied in higher 
education courses such as writing, computer science, arts and engineering, etc (Liu, Lin, & Yuan, 2002). There 
are a number of studies illustrating how this process can be applied in both summative and formative 
evaluations. The majority of the literature on peer assessment in higher education has focused on the 
“assessment of individual contribution to group work” or the correlation between peer rating and instructor 
rating (Hanrahan & Isaacs, 2001, Sluijsmans et al., 2002). There are also some studies exploring the perceptions 
and feeling of students towards this process. 

Peer assessment’s benefits on higher thinking and cooperative learning have been established. Pope 
(2001) suggested peer assessment stimulates student motivation and encourages deeper learning. Freeman 
(1995) argued that studying the marking criteria and evaluating peers’ work can improve students’ critical 
assessment skills. Topping (1998), after reviewing 109 articles focusing on peer assessment, confirmed that 
peer assessment yields cognitive benefits for both assessor and assessee in multiple ways. Those “benefits might 
accrue before, during and after” the process. He further concluded that feedback yielded from this process has a 
positive impact on students’ grades and subjective perceptions. 

Most current peer assessment methods are conducted through paper-based systems. Two concerns 
associated with this system that hinder the widespread acceptance of this process are anonymity and the 
administrative workload. 

Researchers noted their concerns towards the anonymity issue in peer assessment (Davies, 2002). One 
assumption of this process’s credibility is that students usually provide fair and unbiased feedback to their 
peers. However, as reported by a number of studies, students find it difficult to rate their peers. They don’t want 
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to be too harsh on their peers; they are uncomfortable critiquing others’ work (Hanrahan & Isaacs, 2001; 
Topping et al., 2000). Conducted in an open environment, potential biases like friendship, gender or race could 
cause students to rate good performance down or poor performance up. Instructors need to design and maintain 
a distribution system to keep both reviewers’ and reviewees’ information confidential and anonymous, and at 
the same time, traceable for instructors to maintain the fluency of the process. 

Taking more control of their learning process motivates students. Assessing peers’ projects deepens 
their understanding of the topic being reviewed. Constructive feedback from peers helps to reevaluate and 
improve their own performance. These steps all contribute to more professional performance. At the same time, 
one of the advantages of peer assessment is the reduced assessment time for instructors. The instructor will 
spend less time diagnosing the underlying problems of student response, providing feedback, and reassessing 
students’ revisions. This, of course, is good news for instructors who face the pressure brought by continuous 
growth in student enrollment and limited instructional time. However, another problem might be raised at the 
same time: the management of feedback documentation (Davies, 2002). Hanrahan and Isaac (2001) reported 
more than 40 person hours for documentation work in classes with 244 students. The load increases with larger 
classes. This is one of the major reasons some researchers found this process time consuming. 

Technology-mediated peer assessment has been proposed as a solution to provide anonymity and 
minimize the workload. In this system, data can be automated and summarized, and students and instructors 
have instant access to data once they are generated. The whole process can be conducted in an anonymous way 
via the Internet. Reviewers and reviewees are not aware of each other. However, the integration of technology 
in peer assessment in higher education is still at an early stage of development. Limited data are reported even 
though various forms of computer-assisted peer assessment methods have been described (Topping, 1998). Our 
study addresses this issue by investigating an application of a peer assessment process that is delivered via an 
anonymous Web-based feedback management system. Our interest is in the impact of technology-based peer 
feedback on student meaningful learning and students’ perceptions of this method in higher education. 

In this study, peer assessment and feedback were utilized only for promoting learning, not as a 
substitute for instructor grading. Its three critical aspects include: 1) defining assessment criteria, in which 
students think about what is required; 2) evaluating the performances of peers; 3) providing constructive 
feedback for further project improvement. Compared with other methods in this area, this study is innovative 
because it utilizes a database-driven peer feedback website to ensure anonymity, simplify data management and 
stimulate student interaction. 

Based on the outcome of previous studies, our hypothesis is: 

1. Web-based peer feedback engages students in critical thinking and promotes meaningful learning, thus 
improving project quality. 

2. Students gain positive perceptions about this process. They feel the process promotes deeper learning 
and helps them improve their project quality. 

Facilitating Website 

The emergence of information technology and rapid increase of online capacity have provided a new 
arena for education. Like other instructional platforms, innovative methods integrating technology have been 
proposed and tested in the assessment field. In the early 1990’s, a novice collaborative learning network was 
studied at the University of Liverpool, England (Rada, Acquah, Baker, & Ramsey, 1993; Rushton, Ramsey, & 
Rada, 1993). One feature of this multi-user database-driven system was designed for facilitating peer 
assessment. This cost-effective tool constructed an environment where students could easily read, grade and 
provide suggestions to each other’s work. Although this system presented incomparable superiority in 
stimulating students’ interaction and reducing administrative load, as noted by the authors (Rushton et al., 

1993), this process was not anonymous. Assessees’ identities could be easily revealed. 

Tsai and his colleagues (Tsai, Liu, Lin, & Yuan, 2001) employed a peer review network to foster 
students’ critical thinking skills. Students completed their projects and uploaded them to the network. This 
network enabled students to review each other’s performance and provide constructive feedback. Then students 
revised their own work according to the comments from peers. This procedure was repeated two or three times. 
Preliminary observation suggested that this system had positive influence on students’ assignments. Tsai further 
asserted that peer assessment supported by a network was the most effective 

Based on the previous research, our study was designed to ensure anonymity and facilitate the peer 
review process. A database-driven website was built that enabled students to register and log in with the 
username and password they specified. This system contained separate interfaces for instructors and students 
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(Figure 1). In the student interface, each student was randomly assigned two WebQuest projects created by two 
peers. Once students logged in, they could perform two roles — reviewer and reviewee. As reviewers, they 
reviewed the two assigned projects and provided their feedback confidentially according to the marking criteria 
for each project. The data were summarized for the author of each project; as reviewees, they had access to the 
feedback for their own projects. The instructor interface was designed to enable instructors to keep track of the 
peer review process. For each student, the instructor had access to the two reviews created by the student as 
well as the feedback this student’s project received from two peers. 

Figure 1 



Student Interface 




Instructor Interface 




This system has the following major merits: 

1. Anonymity was assured. This system ensured anonymity in two ways. First, students’ identities were 
coded as numbers. No personal information, such as initials of their names, could be associated with 
their work. Secondly, students’ projects were WebQuest web sites. Since they were typed and running 
on the Internet, no handwriting would reveal their identities or characteristics, such as gender. The 
potential risk of gender bias demonstrated by Falchikov and Magin’s study (1997) was eliminated. 

2. Management workload was reduced. All the data were aggregated and transmitted from users’ 
computers to database. Management workload was minimal. 

3. Students’ interaction was stimulated. Submitted data were instantly summarized. Students and 
instructors had immediate access, which encouraged students’ engagement and promoted their 
interaction. 



Methods 



Subjects 

This study was conducted with forty-eight students from three undergraduate classes at a central US 
university. Although two teachers instructed these three classes, the same procedure was followed. Students 
were all from the same course entitled “Instructional Technology” at the College of Education and Human 
Sciences. Students were randomly assigned into an experimental group (27) and a control group (21). One 
student in the experimental group dropped the study for personal reasons. Since this course is a required 
technology application course for pre -service teachers at a college level, students have different academic 
backgrounds and range from freshman to senior. 

Procedure 

In this study, students were asked to build a WebQuest project and upload it to the Internet. A 
WebQuest is “an inquiry -oriented activity in which most or all of the information used by learners is drawn 
from the Web” (Dodge & March, 1995). This model, developed by Bernie Dodge and Tom March in the early 
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1995, is designed to involve users in a learning process of analysis, synthesis and evaluation, which promotes 
their critical thinking and scaffolding skills. 

In the control group, students were asked to individually develop WebQuest projects by themselves 
after studying the content area and the assessment criteria. In the experimental group, the following five stages 
were involved (Figure 2): 

Figure 2 



Experiment Groups Control Groups 



Stage 1 : Studying the content 
area and discussing assessment 
criteria 
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peers and improving projects 






1 






Final Stage: Project submission 
to the instructors 




Final Stage: Project submission 
to the instructors 



Stage 1: Studying the content area and discussing assessment criteria 

After thoroughly studying the content area, students were presented a rubric and were asked to study it. 
Students were informed that this was the evaluation criteria that would be used by the instructor in assessing 
their projects and for their use in reviewing peers’ projects. It depicted the basic elements required for a quality 
WebQuest; thus it was important and beneficial to the assignment. The assessment rubric was studied in two 
levels in a student-centered atmosphere. First students formed groups and discussed the rubric; then they were 
encouraged to share their understanding in class. 

Stage 2: Developing WebQuest project 

Students were requested to make a WebQuest project, build it a web site, and upload it to the Internet. 

Stage 3: Judging the performances of peers and providing feedback 

The website built to facilitate the peer review process was introduced to students. Once students logged 
onto the peer feedback website, they had access to two peers’ WebQuest projects, which were randomly 
assigned to them. Students were asked to rate the projects and provide detailed comments according the rubric 
criteria. 

Stage 4: Reviewing feedback from peers and improving their own projects 

Feedback from peers was automatically summarized and made available to the creator of each project. 
After viewing the peer rating scores and comments, students had the opportunities to go back to improve their 
own projects. 

Final Stage: Project submission to instructors 

Students submitted their projects to instructors for grading. 
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Survey 



After students in the experiment groups submitted their final projects, they were asked to complete a 
survey. Twenty-two students in the experimental groups responded to this survey. The survey replicated from 
previous study (Lin, Liu, & Yuan, 2002) consisted of 11 5-point Likert Scale items dealing with their general 
perceptions about the process, as well as two open-ended questions related to their likes and dislikes: “Please 
specify what you like most in this peer assessment procedure.” “How would you change this peer assessment 
procedure? And why?” 

Scoring Procedure 

Two independent raters were trained and each of them graded all the projects using a rubric (Appendix 
1) with slight modifications from an established rubric by Dodge (2001). Projects were assessed in six areas and 
received a score from 0 to 50 points. Both of the raters were former instructors of this course. They were 
knowledgeable in the content area and experienced in assessment. Furthermore, they were not associated with 
the course or students at the time of scoring, which minimized any potential existing biases. Students were 
instructed to remove any personal information in their projects. Projects from both experimental and control 
groups were coded and mixed together. Raters could not identify individual students or identify which group 
projects were from. 

Inter-rater reliability was assessed for the two raters. The Pearson Correlation between the scores from 
two raters was .680. 



Results 

Two types of data were gathered in this study. The first type compared the difference of student 
learning represented by project quality between the experimental and control groups. The second type 
considered students general perceptions of this technology-mediated peer assessment procedure. 

Difference of Projects Quality 

Each project received two scores from two independent raters. The mean score was calculated and 
awarded to each project. ANOVA was utilized to test if there was any significant difference between the project 
scores of the control group and the experimental group. 

Table 1 



Group 


Mean 


SD 


N 


Experimental 


37.95 


5.54 


26 


Control 


36.57 


7.28 


21 



The difference between the two means (37.95 vs. 36.57) is not significant, F (45) = .545, p = .464. 

Interactive graph (Figure 3) shows the confidence intervals of the group means present a large overlap. 
There is a trend that the mean score in the experimental group was slightly higher than that of the control group 
and scores in the experimental group were located in a more condensed cluster. 
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Figure 3 




1= experimental group 



Students’ Perceptions on Peer Assessment 

Twenty-two students in the experimental group responded to the post-assessment survey. This survey 
consisted of two parts. The first part was an 1 1-item 5-point Likert Scale ranging from 1 (strongly disagree) to 5 
(strongly agree). The second part consisted of two open-ended questions regarding students’ likes and dislikes: 
“Please specify what you like most in this peer assessment procedure.” “How would you change this peer 
assessment procedure? And why?” 

Table 2 



Peer Assessment Survey 





Mean 


SD 


Minimum 


Maximum 


1.1am content with my own work. 


445 


.67 


3 


5 


2. 1 learn more from peer assessment than 
from traditional teacher assessment. 


3.64 


.73 


3 


5 


3. The procedures on how to do peer 
assessment are clearly outlined 


4.55 


.51 


4 


5 


4. Peer assessment is a worthwhile 
activity. 


4.18 


.59 


3 


5 


5. Peers have adequate knowledge to 
evaluate my work. 


3.73 


.63 


3 


5 


6. 1 benefited from peers' comments. 


4.32 


.65 


3 


5 


7. The peers' comments on my work 
were fair. 


4.23 


.75 


3 


5 


8. Peers can assess fairly. 


3.95 


.65 


3 


5 


9. 1 have benefited from marking peers' 
work. 


3.91 


.92 


2 


5 


10 l took a serious attitude towards 
marking peers' work. 


4.64 


49 


4 


5 


11 1 felt that 1 was critical of others 
when marking peers' work 


4 00 


93 


2 


5 
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This table provides a picture of students’ positive perceptions on peer assessment. Students reached a 
general satisfaction level for all of the items. 

For the first open-ended question (“Please specify what you like most in this peer assessment 
procedure.”), three major themes were depicted. First, the opportunity to review and grade peers’ performance 
urged students on to greater efforts in the content area and the marking criteria. Secondly, feedback students 
received from peers helped them improve their projects. The third was the comfort brought by anonymous 
marking and instant feedback. 

For the second open-ended question (“How would you change this peer assessment procedure? And 
why?”), three themes emerged. Several students stressed their satisfaction with this technology -mediated 
process. They stated that they wouldn’t suggest any changes. Some students would have liked more than two 
peers rating their projects. They found it difficult to decide what to do if two peers gave them conflicting 
comments. Some students asked for more critical and constructive feedback. 

Discussion 

This study, investigating the influence of peer feedback in student meaningful learning and exploring 
student satisfaction level of this process, presented us an interesting picture. Data indicated that there was no 
significant difference of the project quality between the control and experimental groups. However, post- 
assessment survey revealed students’ general recognition and acceptance of this process. These seemingly 
contradictory outcomes may be explained in part by the following. 

First, independent ratings were used to compare the difference of students’ project quality between two 
groups. To assure the reliability and consistency of scoring, inter-rater reliability was assessed. Two 
independent raters graded all projects according to the rubric. However, their grading didn’t reach an agreement 
at a satisfactory level (the Pearson correlation equaled .680). Therefore, we cannot conclude that the scoring 
was reliable. There are many possibilities. It could be that our marking criteria was not categorized and 
described well enough for raters to evaluate students’ projects and reach an agreement. Or it could be we need 
to provide more training to raters before they started grading. Or our measurement may not have discriminated 
levels of quality. 

Secondly, Topping (1998) suggested that the benefits from peer assessment could accumulate anytime 
before, during or after the procedure. Peer assessment could have a positive impact on students’ grades; it could 
also aid in the building of transferable skills and the foundation of lifelong learning. Like most peer assessment 
studies in literature, this study only focused on summative evaluation. Though statistically it revealed no 
significant difference of project quality between the control and experimental groups, the general agreement 
students reached in the post-assessment survey suggested students valued peer assessment as a worthwhile 
activity and they benefited marking peers’ work. If formative evaluation was applied, there might be some 
indicators that student meaningful learning is enhanced by this process. 

Finally, in the interactive graph (Figure 3), though the confidence levels have a large overlap and the 
difference between the two groups of scores is not significant, there is a trend that the mean score in the 
experiment groups was slightly higher than that of the control groups. If a bigger pool of students had 
participated in this study, the result might be the different. The variability of the scores in the experimental 
group was smaller than the control group. Further study may reveal that the procedure had a differential impact 
on the lower scoring students. 

Based on these interpretations, we suggest that further study with a larger number of subjects and more 
instructors, and improved quality assessment measures is warranted. 

The merits of this computer- mediated peer assessment process — anonymity and promptness — were 
recognized and addressed in students’ survey responses. One student stated, “it helped out not knowing who the 
person was critiquing my project”, another noted “it probably puts less pressure on the grader.” Students liked 
“the instant feedback” from peers. At the same time, the instructors recognized a significant reduction of 
management workload. All the data were automatically summarized by the system. Students and instructors had 
instant access to data once they were generated. This certainly reduced the administration load. 

Though the difference of project quality between groups was not significant, students expressed a 
rather high level of satisfaction toward this computer-mediated peer assessment process. Overall, we felt the 
peer feedback process in this study was a worthwhile activity. During this process, students were fully engaged 
and they changed their roles from reviewers to reviewees, and then improved their work. During this process, 
students’ interaction was stimulated and their critical thinking skills were fostered. Compared to paper-based 
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systems, a computer- mediated system is certainly promising and provides advantages. 
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Appendix 1 WebQuest Rubric 



Beginning Developing Accomplished Score 

Overall Aesthetics (This refers to the WebQuest page itself, not the external resources linked to it.) 

Overall Visual 0 points 2 points 4 points 

Appeal 

There are few or Graphic elements Appropriate and thematic 

no graphic sometimes, but not graphic elements are used 

elements. No always, contribute to make visual connections 

variation in layout to the that contribute to the 

or typography. understanding of understanding of concepts, 

concepts, ideas and ideas and relationships. 

OR relationships. Differences in type size 

There is some and/or color are used well 

Color is garish variation in type and consistently, 

and/or typographic size, color, and 
variations are layout, 

overused and 
legibility suffers. 

Background 
interferes with the 
readability. 

Navigation & 0 points 2 points 4 points 

Flow 

Getting through There are a few Navigation is seamless. It 

the lesson is places where the is always clear to the 

confusing and learner can get lost learner what all the pieces 

unconventional. and not know are and how to get to them. 

Pages can't be where to go next, 

found easily and/or 
the way back isn't 
clear. 

Mechanical 0 points 1 point 2 points 

Aspects 

There are more There are some No mechanical problems 

than 5 broken broken links, noted. 

links, misplaced or misplaced or 

missing images, missing images, 

badly sized tables, badly sized tables, 

misspellings misspellings 

and/or and/or 

grammatical grammatical 

errors. errors. 

Introduction 

Motivational 0 points 1 point 2 points 

Effectiveness of 

Introduction The introduction is The introduction The introduction draws the 

purely factual, relates somewhat reader into the lesson by 

with no appeal to to the learner's relating to the learner's 

relevance or social interests and/or interests or goals and/or 

importance describes a engagingly describing a 

compelling compelling question or 
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OR question or problem, 

problem. 

The scenario posed 
is transparently 
bogus and doesn't 
respect the media 
literacy of today's 
learners. 

Cognitive 0 points 1 point 2 points 

Effectiveness of 

the Introduction The introduction The introduction The introduction builds on 

doesn't prepare the makes some learner's prior knowledge 

reader for what is reference to or effectively prepares the 

to come, or build learner's prior learner by foreshadowing 

on what the learner knowledge or what the lesson is about, 

already knows. previews to some 
extent what the 
lesson is about. 

Task (The task is the end result of student efforts... not the steps involved in getting there.) 

Connection of 0 points 2 point 4 points 

Task to 

Standards The task is not The task is The task is referenced to 

related to referenced to standards and is clearly 

standards. standards but is not connected to what students 

clearly connected must know and be able to 
to what students do to achieve proficiency 
must know and be of those standards, 
able to do to 
achieve 
proficiency of 
those standards. 

Cognitive Level 0 points 3 points 6 points 

of the Task 

Task requires Task is doable but Task is doable and 

simply is limited in its engaging, and elicits 

comprehending or significance to thinking that goes beyond 

retelling of students' lives. The rote comprehension. The 

information found task requires task requires synthesis of 

on web pages and analysis of multiple sources of 

answering factual information and/or information, and/or taking 

questions. putting together a position, and/or going 

information from beyond the data given and 
several sources. making a generalization or 
creative product. 

Process (The process is the step-by-step description of how students will accomplish the task. ) 

Clarity of o points 2 points 4 points 

Process 

Process is not Some directions Every step is clearly stated, 

clearly stated. are given, but there Most students would know 

Students would not is missing exactly where they are at 

know exactly what information. each step of the process 

they were Students might be and know what to do next. 
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supposed to do just confused, 
from reading this. 

Scaffolding of 0 points 3 points 6 points II 

Process 

The process lacks Strategies and The process provides 

strategies and organizational students coming in at 

organizational tools embedded in different entry levels with 

tools needed for the process are strategies and 

students to gain the insufficient to organizational tools to 

knowledge needed ensure that all access and gain the 

to complete the students will gain knowledge needed to 

task. the knowledge complete the task, 

needed to complete 

Activities are of the task. Activities are clearly 

little significance related and designed to 

to one another Some of the take the students from 

and/or to the activities do not basic knowledge to higher 

accomplishment of relate specifically level thinking, 

the task. to the 

accomplishment of 
the task. 

Richness of 0 points 1 points 2 points 

Process 

Few steps, no Some separate Different roles are assigned 

separate roles tasks or roles to help students understand 

assigned. assigned. More different perspectives 

complex activities and/or share responsibility 

required. in accomplishing the task. 

Resources (Note: you should evaluate all resources linked to the page, even if they are in sections other 
than the Process block. Also note that books, video and other off-line resources can and should be used 

where appropriate.) 

Relevance & 0 points 2 point 4 points 

Quantity of 

Resources Resources There is some There is a clear and 

provided are not connection meaningful connection 

sufficient for between the between all the resources 

students to resources and the and the information needed 

accomplish the information for students to accomplish 

task. needed for the task. Every resource 

students to carries its weight. 

OR accomplish the 

task. Some 

There are too resources don't add 

many resources for anything new. 
learners to look at 
in a reasonable 
time. 

Quality of 0 points 2 points 4 points 

Resources 

Links are Some links carry Links make excellent use 

mundane. They information not of the Web's timeliness and 

lead to information ordinarily found in colorfulness. 
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that could be found 
in a classroom 
encyclopedia. 


a classroom. 


Varied resources provide 
enough meaningful 
information for students to 
think deeply. 




Evaluation 


Clarity of 

Evaluation 

Criteria 


0 points 

Criteria for success 
are not described. 


3 points 

Criteria for success 
are at least 
partially described. 


6 points 

Criteria for success are 
clearly stated in the form of 
a rubric. Criteria include 
qualitative as well as 
quantitative descriptors. 

The evaluation instrument 
clearly measures what 
students must know and be 
able to do to accomplish 
the task. 




Tol 


tal Score 


/50 
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